feat(kubernetes): add sidecar supervisor topology by TaylorMutch · Pull Request #2076 · NVIDIA/OpenShell

TaylorMutch · 2026-06-30T22:11:21Z

Summary

Adds the Kubernetes sidecar supervisor topology after the combined-topology base from #2074.
The default combined topology remains unchanged, and sidecar is opt-in.

sidecar moves pod-level network enforcement and gateway forwarding into a dedicated supervisor sidecar. The agent container can run as the resolved sandbox UID/GID with runAsNonRoot, no privilege escalation, and all Linux capabilities dropped.

This draft intentionally includes the prerequisite UID/GID commits from #1973 for now. We expect to rebase those out after that work lands separately.

Runtime validation status:

sidecar has been validated on GKE Autopilot clusters created with gcloud container clusters create-auto --workload-policies=allow-net-admin; the Google Cloud CLI reference lists allow-net-admin as the supported Autopilot workload policy: https://docs.cloud.google.com/sdk/gcloud/reference/container/clusters/create-auto#--workload-policies
sidecar is experimental with Kata Containers.
sidecar is known to fail with gVisor because it depends on pod-local network rule setup.

Sidecar mode preserves gateway session and SSH behavior, but intentionally runs the process supervisor in network-only mode. Filesystem policy, process privilege dropping, and process/binary identity checks are not applied in this mode.

Related Issue

References #1827, #981, #899, #1305.

Related PRs: #1973, #2074, #2016.

Changes

Include prerequisite sandbox UID/GID support: numeric sandbox identities, configurable Kubernetes sandbox UID/GID values, and OpenShift SCC annotation fallback.
Add the sidecar supervisor topology and the related processEnforcement and proxyUid configuration.
Render sidecar-mode sandbox pods with a network init container, non-root supervisor sidecar, and unprivileged agent container.
Add process-supervisor network-only behavior for sidecar mode while keeping SSH/session relay behavior intact.
Add sidecar e2e Helm values and Skaffold profile support.
Add a sidecar topology row to the branch Kubernetes E2E matrix.
Document topology choice, architecture diagrams, permission model, RuntimeClass validation status, and network-only tradeoffs.

Testing

Checklist

Follows Conventional Commits
Commits are signed off (DCO)

copy-pr-bot · 2026-06-30T22:11:25Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

github-actions · 2026-06-30T22:11:44Z

🌿 Preview your docs: https://nvidia-preview-pr-2076.docs.buildwithfern.com/openshell

github-actions · 2026-06-30T22:45:56Z

Label test:e2e applied, but pull-request/2076 is at {"messa while the PR head is 6276de3. A maintainer needs to comment /ok to test 6276de38ac92b65b0b24a68b8d09a5eed06f50d7 to refresh the mirror. Once the mirror catches up, re-run Branch E2E Checks from the Actions tab.

mrunalp · 2026-07-06T15:29:26Z

Some comments with help of agent

1. Central trust boundary — the workload can use the sandbox's gateway identity .
Client mTLS and the SA token render 0440 with fsGroup=sandbox_gid, the process leaf skips supervisor-identity isolation in network-only, and the 127.0.0.1:18080 forward is always open. Since the workload child also runs with sandbox_gid, it can read those credentials and reach the gateway directly.

2. nftables fence only covers TCP/UDP.
The output chain is policy accept and only rejects TCP/UDP from non-proxy_uid UIDs. ICMP (e.g. unprivileged ping sockets) and other L4 protocols fall through. CAP_NET_RAW is dropped so raw sockets aren't available, but this is a narrower egress boundary than combined mode's per-netns setup. Low severity — suggest a brief comment in generate_sidecar_bypass_ruleset noting the intentional scope.

3. processEnforcement=full does not restore network binary-identity.
supervisor_sidecar_env hard-codes NETWORK_BINARY_IDENTITY=relaxed regardless of process_enforcement, so full restores agent-container Landlock/privilege-drop but network policy stays endpoint/L7-scoped. The docs are technically accurate ("combined-mode process/filesystem controls"), but a reader could reasonably infer full ≈ combined. A one-line note that network policy remains endpoint-scoped in sidecar mode regardless of processEnforcement would prevent that.

4. Doc drift in the debug skill.
debug-openshell-cluster/SKILL.md says the sidecar runs "with primary GID 0" to read the token, but the implementation uses runAsGroup: sandbox_gid + fsGroup: sandbox_gid (which re-owns the projected volume to sandbox_gid). Please update to match.

TaylorMutch · 2026-07-06T20:21:02Z

Central trust boundary — the workload can use the sandbox's gateway identity .
Client mTLS and the SA token render 0440 with fsGroup=sandbox_gid, the process leaf skips supervisor-identity isolation in network-only, and the 127.0.0.1:18080 forward is always open. Since the workload child also runs with sandbox_gid, it can read those credentials and reach the gateway directly.

Addressed this by moving the supervisor and sandbox to run with different GIDs.

Updated the other areas with clarifying comments.

mrunalp · 2026-07-07T00:31:14Z

Deployed this branch on an OpenShift 4.22 cluster (K8s 1.35) with supervisor.topology=sidecar and TLS + mTLS enabled (default PKI init job). The sidecar pod renders correctly, but openshell-network-init is stuck in CrashLoopBackOff:

Error:   × failed to create sidecar directory /etc/openshell-tls/proxy/client
  ╰─▶ Permission denied (os error 13)

$ oc -n openshell get pod test-sidecar
NAME           READY   STATUS                  RESTARTS
test-sidecar   0/2     Init:CrashLoopBackOff   6

Root cause

In run_network_init (crates/openshell-sandbox/src/main.rs) the ordering is:

prepare_sidecar_directory(sidecar_tls_dir /* /etc/openshell-tls/proxy */, proxy_uid, proxy_gid, 0o755) — this chowns the TLS dir to proxy_uid (e.g. 1337), mode 0755.
copy_sidecar_client_tls_if_present(...) → prepare_sidecar_directory(dest_dir /* …/proxy/client */, …) → create_dir_all.

Step 2 runs as root with capabilities.drop: ["ALL"] (the init container only adds NET_ADMIN, NET_RAW, CHOWN, FOWNER, so it has no CAP_DAC_OVERRIDE). The parent dir is now owned by uid 1337, mode 0755, so root is neither the owner nor has group-write → EACCES. Root can no longer bypass DAC checks because DAC_OVERRIDE was dropped.

Note the state dir was already changed to 0o2775 (setgid + group-write), but the TLS dir is still 0o755, and it's the one that gets a subdirectory created in it right after being chowned away from root.

TaylorMutch · 2026-07-07T02:56:31Z

@mrunalp I reproduced locally and pushed a fix. Please have a look when you get a chance!

Add the Kubernetes sidecar supervisor topology, its Helm/Skaffold configuration, topology documentation, and sidecar e2e matrix coverage. Skip root-only sandbox identity rewriting when process enforcement is network-only so the low-permission sidecar process container can start successfully. Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

drew · 2026-07-07T06:11:09Z

 # staging directory before the image build job runs.

-FROM scratch AS supervisor
+FROM alpine:3.22 AS supervisor


Making a note that this might break the podman implementation which expects a scratch container. I might be wrong here though. I would expect E2E tests to shake this out.

drew · 2026-07-07T06:20:01Z

    supervisor_topology          = {{ .Values.supervisor.topology | default "combined" | quote }}
+    process_enforcement          = {{ .Values.supervisor.processEnforcement | default "network-only" | quote }}


Can the supervisor_topology automatically determine how process_enforcement is configured? Configuring topology and process enforcement seems to leak implementation details of the topology.

drew · 2026-07-07T06:22:21Z

+    case "${selected_engine}" in
+      docker|podman)
+        export CONTAINER_ENGINE="${selected_engine}"
+        return
+        ;;
+      *)
+        echo "ERROR: CONTAINER_ENGINE=${CONTAINER_ENGINE} is invalid; expected docker or podman" >&2
+        exit 2
+        ;;
+    esac


why is there a docker branch here?

drew · 2026-07-07T06:28:30Z

This draft intentionally includes the prerequisite UID/GID commits from #1973 for now. We expect to rebase those out after that work lands separately.

Looks like you've rebased now since that's been merged?

drew · 2026-07-07T06:30:13Z

+/// Supervisor pod/runtime topology. Kubernetes sidecar mode sets this to
+/// `"sidecar"`; the default combined supervisor path omits it.
+pub const SUPERVISOR_TOPOLOGY: &str = "OPENSHELL_SUPERVISOR_TOPOLOGY";
+
+/// Network enforcement backend selected by the compute driver.
+pub const NETWORK_ENFORCEMENT_MODE: &str = "OPENSHELL_NETWORK_ENFORCEMENT_MODE";
+
+/// Process enforcement mode selected by the compute driver.
+///
+/// The default when unset is `"full"`, where the process supervisor enforces
+/// filesystem/process policy before spawning workloads. Kubernetes sidecar
+/// topology sets this to `"network-only"` so the process wrapper can run as
+/// the sandbox UID without Linux capabilities while preserving SSH/session
+/// behavior.
+pub const PROCESS_ENFORCEMENT_MODE: &str = "OPENSHELL_PROCESS_ENFORCEMENT_MODE";
+
+/// Whether network policy evaluation must bind requests to the peer binary.
+///
+/// The default when unset is `"required"`. Kubernetes sidecar experiments may
+/// set this to `"relaxed"` to enforce endpoint and L7 policy without per-binary
+/// `/proc` identity binding.
+pub const NETWORK_BINARY_IDENTITY: &str = "OPENSHELL_NETWORK_BINARY_IDENTITY";
+
+/// File written by the network supervisor when sidecar networking is ready.
+pub const SUPERVISOR_READY_FILE: &str = "OPENSHELL_SUPERVISOR_READY_FILE";
+
+/// File written by the process supervisor with the workload entrypoint PID and
+/// read by the network sidecar for process/binary-bound network policy checks.
+pub const ENTRYPOINT_PID_FILE: &str = "OPENSHELL_ENTRYPOINT_PID_FILE";
+
+/// Local protobuf policy snapshot written by the network sidecar and read by
+/// the process-only supervisor in Kubernetes sidecar topology.
+pub const SIDECAR_POLICY_SNAPSHOT_FILE: &str = "OPENSHELL_SIDECAR_POLICY_SNAPSHOT_FILE";
+
+/// Local provider environment snapshot written by the network sidecar and read
+/// by the process-only supervisor in Kubernetes sidecar topology.
+pub const SIDECAR_PROVIDER_ENV_SNAPSHOT_FILE: &str = "OPENSHELL_SIDECAR_PROVIDER_ENV_SNAPSHOT_FILE";
+
+/// Optional TLS server name override used when connecting to the gateway.
+pub const GATEWAY_TLS_SERVER_NAME: &str = "OPENSHELL_GATEWAY_TLS_SERVER_NAME";
+
+/// Directory where the network supervisor writes the proxy CA files consumed
+/// by workload child processes.
+pub const PROXY_TLS_DIR: &str = "OPENSHELL_PROXY_TLS_DIR";
+


why do these constants need to be in core?

drew · 2026-07-07T06:32:24Z

+            install_sidecar_iptables_legacy_bypass_rules(proxy_uid).map_err(|iptables_error| {
+                miette::miette!(
+                    "sidecar nft ruleset load failed: {nft_error}; sidecar iptables-legacy fallback failed: {iptables_error}"
+                )
+            })


(from agent review)

When nftables loading fails on a dual-stack pod, this fallback reports success after installing only IPv4 iptables-legacy rules. IPv6 TCP/UDP traffic remains unrestricted, allowing workloads to bypass the sidecar proxy and network policy entirely. Install equivalent ip6tables-legacy rules or fail closed when IPv6 cannot be fenced.

TaylorMutch mentioned this pull request Jun 30, 2026

feat(kubernetes): add proxy-pod supervisor topology #2077

Open

9 tasks

TaylorMutch mentioned this pull request Jun 30, 2026

feat(kubernetes): add cni-sidecar supervisor topology #2078

Draft

8 tasks

TaylorMutch added the test:e2e Requires end-to-end coverage label Jun 30, 2026

TaylorMutch marked this pull request as ready for review June 30, 2026 22:46

TaylorMutch requested review from a team, derekwaynecarr, maxamillion and mrunalp as code owners June 30, 2026 22:46

TaylorMutch force-pushed the feat/kubernetes-sidecar-topology-v2 branch 2 times, most recently from 4f981c2 to 8c9ae53 Compare July 2, 2026 21:12

TaylorMutch force-pushed the feat/kubernetes-sidecar-topology-v2 branch 2 times, most recently from e00a00b to ed858eb Compare July 6, 2026 18:52

TaylorMutch added 9 commits July 6, 2026 21:01

fix(supervisor): avoid similar process id names

566cf14

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

fix(supervisor): avoid similar process id names

9a16c6b

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

fix(sandbox): avoid similar proxy id names

ae3d3d8

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

docs(kubernetes): clarify sidecar topology limits

53a5b90

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

fix(kubernetes): keep sidecar process leaf capless

4904e44

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

fix(kubernetes): refresh sidecar provider env snapshots

e492342

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

test(supervisor): align hot-swap identity regression

8e6c504

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

fix(kubernetes): stage sidecar mtls files before proxy chown

254cdbe

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

TaylorMutch force-pushed the feat/kubernetes-sidecar-topology-v2 branch from 3d80066 to 254cdbe Compare July 7, 2026 04:01

drew reviewed Jul 7, 2026

View reviewed changes

		supervisor_topology = {{ .Values.supervisor.topology \| default "combined" \| quote }}
		process_enforcement = {{ .Values.supervisor.processEnforcement \| default "network-only" \| quote }}

Uh oh!

Conversation

TaylorMutch commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Jun 30, 2026

Uh oh!

github-actions Bot commented Jun 30, 2026

Uh oh!

github-actions Bot commented Jun 30, 2026

Uh oh!

mrunalp commented Jul 6, 2026

Some comments with help of agent

Uh oh!

TaylorMutch commented Jul 6, 2026

Uh oh!

mrunalp commented Jul 7, 2026

Root cause

Uh oh!

TaylorMutch commented Jul 7, 2026

Uh oh!

drew Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

drew Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

drew Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

drew commented Jul 7, 2026

Uh oh!

drew Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

drew Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

TaylorMutch commented Jun 30, 2026 •

edited

Loading